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METHOD FOR SECURE FUNCTION As,the maximum length of the overflowin g data siri ng can 

EXECUTION BY CALLING ADDRESS '6e^nIy'lhT'ciirre^e ^-of-the -stackrthe-inserted-aftack 

VALIDATION ceode'snould'be short in te rms of code4ength. Writing data 

outside-the'stack limit will result in an-exceptionxonditibn 

BACKGROUND OF THE INVENTION s CMa^wil^ P revent-mc-at tock-code- .to-exccvte. Therefore, the 

The present invention relates generally to a method for buffer overflow attacker will be forced tcMe-short^de 

detecting and preventing unauthorized or illegal access Jave^igeJiigWe^ 

attempts within a computer system. More specifically, the Su^Jkwill later be^uhhgedlta-gatn-non-authonzed 

present invention relates to a method tor detecting and ^ — — r -~- — ,~ .« . „ 

** *♦ t t <r — ,-.i- fU u fF , Za^a m Several strategies that attempt to resolve the buffer over- 
preventing attempts to exploit_the^Duffer_overflow-related io ^ . ^ 

, • : * — now weakness are known in the art. One such strategy is to 

weakntite^^ design a compiler designed to prohibit a computer program 

Modern computers are designed according to the require- f rom wr iting past a stack segment array. Another strategy is 

ments of high-level programming languages. A fundamental to detect buffer overflow vulnerable programs off line and 

technique for the structured design of computer programs, alert the user to the possibility that the system privileges may 

associated with the high-level languages, is the procedure or 15 be compromised. 

the function. Another known strategy is using a repair program. The 

Procedures or functions are computer programs. A pro- repair program can repair or fix those vulnerable programs 

cedure call or a function call is a high-level programming that can be used to exploit the buffer overflow weakness, 

concept that modifies the flow of the calling program None of the above provide a method and apparatus for 

execution. In contrast with the more traditional "jump" or 20 prevention of buffer overflow through controlled execution 

"goto" instructions, which also alter the flow of execution, of system or other calls within a computer system, 

a procedure or a function, after the execution of its own SUMMARY OF THE INVENTION 

code returns control to the instruction immediately follow- Qne of ^ ^ g a ter 

ing the call. To implement procedure or function calls in the ^ ai / ti compute r platform that 

manner described a memory device called a stack is utilized. a kemd space aQd a prQcess space The process 

A stack is a contiguous section of memory containing spacxrincfudes--a-us&r~applicationro perative-to-~in tercept 

data, the size thereof is dynamically adjusted by the oper- .s^tenf^lsTru^ 

ating system routines at run time. The data is inserted to and securezfimctioh-executionTThe method of secure function 

removed from the stack by the Central Processing Unit 3Q execution comprisenfie^steps ofexalnining^ interxepted 

(CPU) utilizing Assembler language instructions such as c S y S tejn-cdl-vaUdkty-b^ 

"push" or "pop." c>U-oril3na^ 

The stack contains related information units such as a^dresses-a^ciatedjwimJhejTO 

logical stack frames or Procedure Activation Records that Cce pted-system-caU r miginale~d. 

are inserted therein when a function is called and removed 3S A second aspect of the present invention regards a com- 

when the function returns control to the calling program. p U t er system running an operating system platform, the 

The stack frame itself contains parameters to the called operating system including a kernel space and a process 

function, local variables, pointers to recover the previous space. The process space includes a user application running 

stack frame, saved values of the CPU registers, and the in process space, t thej^^rzappHcation:is:operative-to^ifIter- 

return address of the calling computer program. The return 40 fcep Ulibra ry calls. A method of secure function execution 

address is the instruction pointer of the calling program at ex amines-me"mtercepted4ibraj^-caU-vaha^ y^yjpomparing^ 

the time of the function call. the^ejrepJe^Utontf^ of 

Induced buffer overflow or buffer overflow attack is proclisl^^lid^afesses-assom from 
known in the art. Buffer overflow attacks take advantage of wKich'm^inferceptFd^ibrary-cairaigin^teld^ 

the lack of bounds checking on the size of input being stored 45 A third aspect of the present invention regards a computer 

in a buffer array. Arrays are predefined allocated memory system running an operating system platform. The operating 

devices within a computer system. An attacker can make system includes a kernel space and a process space. The 

erratic changes to data stored adjacent to an allocated array process spa^e.in^hmAS-a-userju^^ running in process 

by writing data intentionally past the end the array. The most space. The^user application is~op e ra tive~ to- s ystem-and , 

typical data structure to be corrupted in this fashion is the 50 ^lun ction-calls ^system-or-functi^ 

stack. Therefore this type of attack is also known as stack melfio^^f-secure-ful^tid^ 

smashing. c the:steps^f-recemng-a-caUer:ro^ 

The prevalent form of buffer overfl ow ex ploitation is to process-memoixjevice-and-de te^ 

attack buffe_r_s_allocated.on_the stackrOne - objectiVe~of-sucb Croutineraa*dress z is-valid4>y^con^^ 

» <anzattack"is"ihserting arwttack-code^inithe-formzof an 55 c address:with::aiproccss^ah^^ 

cexecla^hle^ob'je^t^odezn ative-to-the-aUacked- machineT? A fourth aspect of the present invention regards a com- 

Another objective is to change the return address to point to puter system running an operating system platform. The 

the attack code now residing on the stack. Such attack code operating system includes a kernel apace and a process 

may be utilized to gain enhanced privileges over the com- space. The processspa,ce-includes.a.user application running 

puter system. 60 ir^process space. ThCuser applicltionis-operativeHp^system 

The programs that are attacked using this technique are and-f^cnon caUsrsystemio r-mnction-cal ls -interce pted^^d 

usually high privilege utilities or daemons that run under the ^^--a^emadiof^cu^ 

user-id root to perform essential services. The effect of a C^ prises-the-st e ps of-receivin g-a^caUer-routme-ret urn-addre ss^ 

successful buffer overflow attack is to provide the attacker c^ from-sai d:proc^:memq^^ 

non- authorized root privileges. Gaining root privileges femeicaflen routmeTalitossi 

within a computer system allows non-authorized users ^-routioie^ddress-wim-an-assoti^ 
access to privileged resources. <^_area. 
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Each of the above aspects of the present invention may 
provide the advantage of identifying unauthorized or illegal 
access attempts to software objects within a computer sys- 
tem. 

Each of the above aspects of the present invention may 
provide the advantage of preventing unauthorized or illegal 
attempts to open, process or delete software objects within 
a computing environment. 

Each of the above aspects of the present invention may 
provide the advantage of preventing attempts by potential 
intruders to exploit the buffer overflow-related weakness 
within a computing environment. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The accompanying drawings, which are incorporated in 
and constitute a part of the specification, illustrate preferred 
embodiments of the invention and, together with the 
description, serve to explain the principles of the invention: 

FIG. 1 is a block diagram of the Secure Function Execu- 
tion System environment generally referenced to as system 
100; 

FIG. 2 is a high-level flow diagram of the Secure Function 
Execution Server 116 operation; 

FIG. 3 is a high-level flow diagram of the operation of the 
Secure Function Execution Server initialization module 
referred to in FIG. 2; 

FIG. 4 is a high-level diagram of Secure Function Execu- 
tion Server or the like response to an intercepted system call 
referred to in FIG. 2; 

FIG. 5 a high-level flow diagram of the operation of the 
Secure Function Execution Server and the like library call 
response module referred to in FIG. 2; 

FIG. 6 is a high-level flow diagram of the operation of the 
Calling Address Validation Routine module; 

FIG. 7 is a high-level flow diagram of the Calling Address 
Validation Routine module relating to an another embodi- 
ment of the present invention. 

DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENTS 

The Buffer Overflow event is a common weakness of 
computer software systems related to the fact that software 
developers around the world, utilizing prevalent state-of-art 
programming languages such as "C", have the responsibility 
of handling memory allocations themselves in order to 
accomplish tasks involving specific functionalities of certain 
important applications. The developers may make accidental 
mistakes when allocating memory thereby introducing 
severe programming errors into the computing environment 
and even effect catastrophic operating system failures. 
Furthermore, intentional exploitation of the buffer overflow 
weakness enables sophisticated intruders to gain access to a 
computing environment by passing invalid parameters to a 
faulty program, and consequently make the computer 
execute the code passed as a parameter by the intruder. The 
attacking code could readily contain malicious instructions 
that could produce serious mysfunctions in the computer 
system thus penetrated. 
^-£~£nie-pr_eseittC^^ 

preven^buffer-over£owlev^ts-m-real~time. The method is 
implemented by using the System Call and the API Inter- 
ception System model and thereby attaching a segment of 
custom -written code to each system call and library call to 
investigate the original memory address of the call. Thus, the 



presen t invention overcomes the disadvanta ges of the prior 
artT)y"providing:a:noyel:method;2wJ^ 
atteTnpt-to^e'^loitZthirr^^ 
^progress-by 7 ^^!!^ 
5 cis^co rn injpin' f ronTa" valid address space^ofcthe^calling 
c process. If the call has arrived from an invalid address space, 
a~^e~de]3ugfibnicou^ 
attempt^rOwffer_oyerflow_attack. 

FIG71"is a schemaQclllustration of the system environ- 
ment wherein the Secure Function Execution System is 
operating, generally referred to as system 100, in accordance 
with a preferred embodiment of the present invention. 
Secure Function Execution system 100 of FIG. 1 is designed 
to identify and prevent buffer overflow attacks in real time. 
Secure Function Execution System 100 comprises four 
principal components: 

One) Secure Function.Exe^ut_ign_Server 116 is an active 
component^ ecure~Fulictlon~Execu ti oirServer 1 16 is 
the-operationalcenterof theSecure-FunctionExecution 
System^ lOO. S ecure_Eunction_Execiition-Seryerrll6 
mcojponites^the^^ , 

^ystem-modeLJSecure Function Execution Server 116 
bads ^dcontrolsS^terr r^alHnteire pdon-Gomponerit 
124 r loTds-andrc^uMs7AP05te^ 
^140^an^:r46:mai3£ejxlements ofthe-APTInterception 
^g^~ model 7-;m^ ^hiCh^t hrough- API-Inte rcep tion 
^oj ^l-Mo~dule -lll,-respondXtojyarious-systemIand 
librlu^-calls-an^-operate s^ 
c — user* T^e^Secure-Function^Execution-Server— 116 is 
C^adedu^o~th e_~user~s pac e~r^ a 

computensystem. 
Two) API Interception Modules 134, 140, and 146 that are 
elements of the API Interception System model are 
active components. API Interception Modules 134, 
140, and 146 are Dynamic Link Library (DLL) 
modules, which are loaded by API Interception Control 
Module 111 into each active process address space 118, 
120, and 122 mapped into user space memory device 
112. API Interception Modules 134, 140, and 146 
inside active process address spaces 118, 120, and 122 
are linked wiUTAEi:intefceptionTC6ntrol Module 111 in 
Secure Execution Server 116. The link is established 
directly by API Interce ption Modules 1 34, 140, and 146 
after sa|d;APiririterception-Modules-have_bee^loaaed 
^ into -activ e address spaces 118, 120, and 122. It will be 
appreciated by those skilled in the art that the number 
of API Interception Module copies present within 
active processes residing within the computer system 
address space is associated with th e number of p ro- 
c cesses active at any given time. API-Interception-Mod- 
ures"consist^ofraTDijpatchTRolitmera"Hook~and'Patch 
Q^tineTXI^Enlr^Rduti 
Ctine. The routines are elements of the API Interception 
System model. The description of the functionalities 
and operations of the routines are set forth hereunder. 
Three)^ysTeln7 Call~Inte ice^ 

* activ^c^m^onent. System CairinterceptiorrCompo - 
nenl j.24 o perates in the kernel spa ce memor y device 
114~and-is4iruW4oTSecure-Execution-Gontroi^erver 
116 ~ pjesen Cwit mrEuseT sp ace_. memo ryTdevi ce - 112 . 
Four) API routine 132, 138, and 144 are passive compo- 
nents. API routines 132, 138, and 144 are potential 
objects upon which Secure Function Execution System 
100 might operate. API routines 132, 138, and 144 are 
loaded into process address space memory device 118, 
120, and 122 mapped into user memory device 112 of 
system 100. 
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The subject of the present invention is the secure Function It would be easily perceived that the method of double 

Execution Server 116. Secure Function Execution Server patching is not limited to API functions. Any operative 

116 is utilizing the mechanism of the APT Interception routine in the computing environment could be handled by 

System model to accomplish the present invention's objec- sa id method. 

tives. For the clarity of the disclosure, the reasons for 5 CAfteribemgao^edli^n^^ 140, 

utilizing the API Interception System model will be ^ 146 Hook and Patch TQuXinCy by mcans of the above 

described next, mentioned double patching method, proceeds to hook all 

s As the maximum length of the overflowing strmg can m roudnes m 13g and U4 and tQ tch (hc object code 

oriy bej^Jong-^^^ ^ a if^pr_76utines 132 r l38rand-144'presenrin the process 

j^T? ov^ow-at^ 30 STddress-space-118^ 

. JsXlft length. Any -attempts-to-wnte-into-the-memory-device . . , — : — 

<S5te^ti^ ^ Une 15 c f lled u ^ P atch f acts ** a red f f on ^ m f**L 
tifi^SMSOi^^ Subsequent^ the flow of execution of the patched API 
Qode. Th^efore r the-lack.of-potenu^ routlDe 132 > 138 > and 144 15 ^directed into the Dispatch 
1ul^k:forces:the^ routiDe that 15 P art of me ^ Interception Module. The 
m^erraszto^use^th^Ei^Oe^lEAPI rfuncUops. For 15 Dispatch routine in association with the Pre-entry and the 
example, an attacker that will try to create a file on the local Post-entry routines functions as a pre-processor and a post- 
File System will not write assembler code to accomplish the processor to the API. For example, tests could be made to 
tplrbut"willcaU~a-system : c^l:oO:Ubrai7ifunction:(APIp determine-whether-the-caUing,proffla m-has-me~suita ble 
thaf^ll^expand-into-theT^mta^ privileges toexecute-the A P£:whether;the^rguments;p,assed 
Additio^a^y^me-attacker-mustuse-librar y -functions (API's), 20~b^toecaning-program-arele]pl:or:™^ 
asCitHs^ ^tical ly-impossible^ Cactions:a"ssc^iate~d3 yitfct 
calFpr^hb^^ Interception su^h-j^stoppifig^h^ 
<x p^ if S^^mTnlTdermterceptsrprc-prne^s^ argulnenls^passeB-^^ 

\ aU:ubTaty:ftmcyo^^ ^Ee^AEI. Wh^the^I^tisnelTth^^ 

0*^ I rffimlngiofitheitilegall^ 25 a -jtandar djiexecution~fe76ty 

* Y Csuitable~for the~purpose3f:buf fer"oy erAov^ttacl^identifi- ^flow-of-executiblTis-redirected^ 

^ cHibliJa¥dIpiv^tionj) Therefore, although the API Inter- ClThe~tests_ performed by the Secure cFunctionTExecution 

ception System model is not the subject of the present System J(^ar^p_erf5™ 

invention, in order to provide a thorough understanding of routine-of-the-API-Interception-Mo ^ 

the invention a general description of the API Interception 30 each _ time_ an _ API liiO^aUecirTCere fo re , the investigation 

System model will be given nextPIF^ ll-be-easil ylp.erceived regarding the calling memory ad dress o f the API function in 

^haj ^the-p referred-locj^qn^ oroVrtojdenuTy-and-prevert pfK.*s§-\ 

memory-address-of-a-liblary-ca^l:is^^in_a.routme-mat^ preferabl^in^lullelirinTanzcustomic oded-extensionj -of the\ . C(fA buw^ - 

<actmg-as-a-pre-processor4or 4he-c^ed4 ^ Cjte-gnjrxrcu^ 140, J v v 

IntercepHon[[System riiddel~provides^uch-a-pre = pro L cessor 3Sj* and*"j 46. 

rouline^which^wIU- Ifciwouldib^e'asflycperc^^ 

^wiS-the AEFIntercept ion-System-mo del. It will be also clear Syslemrand^the-presentrinventb^ the 

that the best-placed element to prevent the execution of an handling of SystemrDld^ibutiare^ppJ^We^airDLI^s that 

^gd:caU:is:the^API-InterceptiorrContTol:M6dule 111. <^include-exported?functions. 

In the preferred embodiment of the present invention, the 40 It will be appreciated by those skilled in the art that the 

iffl-Intercepuon^yj ^m-m inter- present invention may operate under various similar 

^ceptiibrary clillsrrjerformed-in-th^ systems, and that the system shown herein is an example to 

^Interception ^Module-^, l^O^-and^^synj^ted^by^PI further illustrate the working of the present invention. 

Inter^Uon-&)n! roFM5~d rt Referring to FIG. 2 there is provided a high-level flow 

space-118r-120,":and:r22:each:time:a new:procesXassociaTed 45 diagram of the Secure Function Execution Server 116 opera- 

wiuVaruXddrcssTspa^ tion. The operation of said Secure Function Execution 

A technique called double patching is utilized to handle an Server 116 is now briefly explained, 

intercepted API function. The method of double patching SecuTe^FiinctionrExec ution (hereinafter re ferred to as^\ ORvMA- 

makes available the option of modifying the API function's SJ^)^ery_er„tt-Oni^ \ 

flow of logic thereby enabling the execution of user- 50 Consequently SFE—Seiver—ll^-commences-its^Tunttime 1 

developed object code segments in association with the QTperationiiirstep 152'-by^nstanlly : mb^it6nhg-system-calls I 

execution of the API function. The objectives of the double Cmaide^b^^ / 

patching method are: (1) To enable the execution of a splen^ffi / 

user-developed routine before the execution the API (step454) ^s des^r3^ m:detail in^FIGr4rSFE Server is also / 

function, (2) to enable consequently the execution of the 55 const!ntly-monitormg4iBr^ 

entire API function, (3) to implement the method seamlessly ^tiolTtliatl^inite 

and transparently in a multi-processing environment, (4) to ^Serviej^esppna^^ 

accomplish the above objectives in regard to all native-code as^describ^drinidetailTih EIGT^S. 

API functions, (5) to accomplish the above objectives with- For reasons related to performance values of the system 

out the necessity of modifying the source code of the API 60 such as the speed of operations, the amount of system calls 

function, therefore without the need to re-compile the API and function calls to be monitored should be preferably 

function. mmimized L 3y^anjdy^ng:me ; Se^^ 

The detailed operation of the double patching method and ,me- potenM:buifenoyer^^ 

the computational logic necessary in order to achieve these ri^de:regar^rig:aCTificant3f^ 

objectives are specifically described in the related patent 65 thatlp referabl v-shouldi>e.monitored. The intruder could try 

application, Attorney Docket No. 253/068, U.S. Ser. No. two types of aU^ksT(A)TScalTttacks in which the attacker 

09/651,395 filed on Apr. 28, 2000. will attempt to open a software object such as a file the 
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access to which requires special privileges that the attacker 
does not possess. As a result, the operating system will 
prevent the access. Consequently, the attacker will attempt 
to change the privilege database of the operating system in 
order to get the special privileges. (B) Remote attacks in 
which the attacker will attempt to execute some code on a 
remote machine in order to create a local file or to try and 
open a remote shell. There are some special system calls and 
system APIs that serve as general use functions and there- 
fore will be called in order to accomplish almost every task. 
The critical system APIs are operating -system-specific. For 
example,jund er th e Windows NT operating system at least 
the-following~runcti 

£) Kernel32.dll— GetProcAddress(..); 
'2) Kernel32.dll— LoadIibraryA(..); 
3) Kernel32.dll— LoadIibraryB(..); 
4j Kernel32.dll— LoadIibraryExA(..); 
5)\Kernel32,dll— LoadIibraryExW(..). 
FIG. 3 illustrates a high-level flow diagram of SFE Server 
116 start-up operation referenced as step 150 of FIG. 2. First 
SFE : SeWer:116;load£S^ 
124-mto:kem el-space~me~m^^ 
e^tabhshmglc^mumc^^ 
Gompppent-l ^SFE-Serrer-H6 -q ueri^ ^ 
ceptic^jC6ln ponenrl24 for the~list-of-activ e-processes 118, 
I20;^^d:;122z(step~186 )~Usm g -tte 
processes 118, 120, and 122 SFE Server 116 creates a lisTof 
valid address ranges for each active process 118, 120, and 
122. Hereinafter this structure will be referred to as Process 
Valid Address Range List. 

A valid address range is defined as a memory execution 
area into which the original file that builds the process was 
mapped. Additionally, a process valid address range includes 
all other virtual address spaces that the process supplemen- 
tary elements are mapped into, such as shared libraries 
(DLLs). 

Consequently, Process Valid Address Range List holds the 
address range into which the process 118, 120, and 122 were 
mapped. Process Valid Address Range List also holds all the 
virtual address ranges into which diverse Dynamic Link 
Library (DLLS) 130, 136, and 142, were-mapped. A 
Dynamic Link Library is a set of callable subroutines finked 
as a binary image that can be dynamically ^ loaded^ by 
applications that utilize mem. c Finallyr'SFE~Server--116 
instru^APHnt^^ 
£lrlS£rcepfi^ 
ceSei:iI8,-:i20, aMX22l(step~190)n 

Turning now to FIG. 4 which^a-highrlevetflowrdiagram 



Module 111 to insert API Interception Module 134 to the 
newly created process address space 118 (step 168) and 
updates Process Valid Address Range List (step 170) by 
adding said process address list to Process Valid Address 

5 Range List. If and when it is determined that the system call 
is of the type of process termination (step 171), SPE Server 
116 updates Process Valid Address Range List (step 170) by 
removing said process valid addresses range from Process 
Valid Address Range List. Control then returns to SFE 

10 Server 116 monitoring library calls in step 156 of FIG. 2. 
FIG. 5 illustrates a high-level flow diagram of the SFE 
Server response to an intercepted library call referred to as 
step 158 of FIG. 2. SFE Server 116 determines in step 172 
whether the library call detected is an illegal call. SFE 

15 Server determines whether the library call is valid by 
comparing the library call originating memory address with 
the range of Process Valid Address List associated with the 
process from which the library call originated. If an illegal 
call is detected SFE Server 116 optionally terminates the 

20 illegal library function (step 180). Alternatively, SFE Server 
116 notifies a user (typically the System Administrator) (step 
182). Altel^ively^SFE:SelyCT user 
predetermmed,or~instructe3-action-(step 1 82) . 
"Tf the library calLcletected is legal (step 172), SFE Server 

25 116 determines if the library call is of the type of DLL 130 
load (step 174). If the decision in step 174 is affirmative, 
then SFE Server 116 instructs API Interception Module 134 
through API Interception control Module 111 to hook and 
patch the library functions (APIS) 132 existing within the 

30 loaded DLL 130 (step 177). After hooking and patching, the 
API Interception Module 134 already loaded into the asso- 
ciated process U8 r is^ow^ope^ive" tg3nte rcept3c^lsTmade 
to^thHibnir^^ 
*is^£theptyperI3Lfc ^ 

35 Valid^AdSressrRangeHto 

addressraage-into-process-Valid-Add ress-Ran ge-List.-If-the 
Siecisionurrsj^ 

i f-the-i ntercepted library call iifof "thFtype DLLrunload (step— 3 
cl7.6)r^Vhenjit-is determined tKafth^libraiy"call-is:of-theJype~ 
4Q_cJ~DIX^uliload-SFE~Seiver"updates~the - Process^Valid 
Addr^ss~R ange-LisCbyrdeleti^ 
ErocesslValidrAddress-Range-Ust (step 178). 

FIGrJ^ifiustratesXfa^ of the opera- 
tion ojjte Calling. Address .Validati on Routine moduk /The 
45 Callrng -Addre ss~Validafion^Routine~module--operates^in 
colijunctionJvritnTAPO^ rou- 
tine: ^ ^ ^ 

'Pre-Entr^-routine^maybe'activate ^^hen zanrAPIjgj^or 
MKeTIlike;iqf^I0~l"isTm^ 



CT:Sffi:Server-lI6,resppnseJo_an_inte^ ■S ystem~l Q0rPfe;EntrCT 



referred.to-as sJepjIjsfM 
^in-ste p-l^O-whether the'systemxalldetectedis-anille gal call 
(oriarlegaT^eall. SFE Server 116 determines whether said 
system call is valid by comparing said system call originat- 



Rojitfii ejnodu leTis^xecuung to 
validate^therAPMi^ 
c (c^llel^Roul5ie^~Gallel^R^^ 
^cationrProgra^lntetfaceT^ 



/ 



ing memory address with the range of Process Valid Address 55^caH:and:the-like. 



Range List associated with said process from which said 
system call originatedrlfan .ineg al calH va^eje^ej^e SFE 
>Server-116- ma y-terminate -the-file gaLf\mcti g^(step 164). 
AUeraatively^SEE_Server~116-may"notify"a'jis^(typically 
the S^telrnVdministra^ 166). 
Alternatively, SFE Server l 116 may perform:another or other 
\ ^seriesroLuserTprecletermined-actions (stepTl66)r-~^ 

If the system calUleTected%-legaL(step-160) SFE Server 
116 examines the said system call to determine whether the 
type thereof is process creation (step 162), If and when 
determined that the system call is of the type of process 
creation, SFE Server 116 instructs API Interception Control 



60 



65 



iCallingrA ddress^vafid ation Routine module commences 
its operation by reading tt^caller^Roufine^return-address 
from-th^ProcTdurc:A^ 

is^n;me-user-stack : segmejit-to is 
a-dy namic-area-of-the ^^ocess~stack-seg ment-used-) as a 
control area for function callsTThe process stack segment is 
a dynamic area of memory belonging to a process. In step 
192 the caller Routine calling address is calculated, and with 
the help of the data in Process Valid Address Range List, it 
is examined if the caller Routine calling address is within 
valid address range limits (step 194). In step 196 it is 
determined whether the calling address is valid or non-valid. 
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To calculate whether the caller Routine calling address is 
within the valid address range limit the caller Routine 
calling address is matched with the valid address range limit. 
If the caller Routine calling address is within the valid 
address range, then the caller Routine calling address is 
valid. Next, Calling Address validation Routine module, by 
Pre -Entry routine or the like, notifies SPE Server 116 or the 
like- about-the test-result^step^l^and.step^OO).-^^^^ 
^It-wiUJ^ppj^ciated by persons skilled in the art thaUrT) 
tm^^illusjr^d^embodiment of the present invention any/ 
unaumorized orrillegatsystem.calto 
form-memoiy^areas^out-of„acUve_process address~space 
memqrj device"^ 

and optionally their execution^ will-be-preyented_byj SEE 
System 100. 

For reasons related to the performance values of the 
system it would be advantageous to reduce the number of 
comparison operations executed by the pre-processor rou- 
tine. Such a reduction could be accomplished readily as the 
known type of buffer overflow attacks typically exploits the 
stack buffer overflow weakness. Thus, it would be advan- 
tageous to determine in one compare operation whether the 
call is coming in from an invalid execution area such as the 
stack. 

Accordingly, reference is now made to FIG. 7, which is a 
high-level flow diagram of the Calling Address Validation 
Routine module relating to an another embodiment of the 
present invention. 

In the embodiment thereof, Calling Address Validation 
Routine module commences its operation by reading the 
caller return address from the Procedure Activation Record 
(stack frame) on the process stack segment (step 202). It will 
be appreciated that reading caller Routine return address is 
significantly faster and more accurate. In step 204 the caller 
Routine calling address is calculated, and it is examined with 
the help of system-level structures to determine whether the 
calling address is inside the address limits of the process 
stack segment (step 206). Such determination is accom- 
plished by comparing the caller Routine calling address with 
address limits of said process stack segment. Next, Calling 
Address Validation Routine module by Pre-Entry routine or 
the like notifies SFE Server 116 or the like about the result 
of the examination (step 210 and step 212). 

It will be appreciated by persons skilled in the art that in 
this further embodiment of the invention any unauthorized 
or illegal system call or library call originating from the 
process stack segment structure will be detected and option- 
ally prevented by SFE System 100. 

Additional advantages will readily occur to the person 
skilled in the art. The invention, in its broader aspects is, 
therefore, not limited to the specific details, representative 
methods, systems and examples shown and described. It will 
be further appreciated by persons skilled in the art that the 
present invention is not limited to what has been particularly 
shown and described hereinabove but rather by the claims 
which follow. 

We claim: 

1. In a computer system running an operating system 
platform, said operating system including a kernel space and 
a process space, said process space including a user appli- 
cation running in process space, said user application opera- 
tive to intercept system calls, a method of secure function 
execution, said method comprising the step of: 

examine said intercepted system call validity by compar- 
ing said intercepted system call originating address 
with range of process valid addresses associated with 
said process from which said intercepted system call 
originated. 
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2. The method of claim 1, further comprising the step of: 
providing notification as to the validity of said intercepted 

system call. 

3. The method of claim 1, further comprising the step of: 
terminating said intercepted system call. 

4. The method of claim 2, further comprising the step of: 
terminating said intercepted system call. 

5. The method of claim 1, further comprising the steps of: 
responsive to process creation inserting application pro- 
gram interface interception module into said created 
process; and 

responsive to process creation updating process valid 
addresses table. 

6. The method of claim 1, further comprising the step of: 
responsive to process termination updating process valid 

addresses table. 

7. In a computer system running an operating system 
platform, said operating system including a kernel space and 
a process space, said process space including a user appli- 
cation running in process space, said user application opera- 
tive to intercept library calls, a method of secure function 
execution, said method comprising the step of: 

examine said intercepted library call validity by compar- 
ing said intercepted library call originating address with 
range of process valid addresses associated with said 
process from which said intercepted library call origi- 
nated. 

8. The method of claim 7, further comprising the step of: 
providing notification as to the validity of said intercepted 

library call. 

9. The method of claim 7, further comprising the step of: 
terminating said intercepted library call. 

10. The method of claim 8, further comprising the step of: 
terminating said intercepted library call. 

11. The method of claim 7, further comprising the steps 

of: 

responsive to system call loading dynamic link library 
hooking and patching library routines associated with 
said dynamic link library; and 

responsive to system call unloading dynamic link library 
updating process valid addresses table. 

12. In a computer system running an operating system 
platform, said operating system including a kernel space and 
a process space, said process space including a user appli- 
cation running in process space, said user application opera- 
tive to system and function calls, said system or function call 
intercepted, a method of secure function execution, said 
method comprising the steps of: 

receiving caller routine return address from said process 

memory device; and 
determining whether caller routine address is valid by 

comparing said caller address routine with process 

valid address table. 

13. The method of claim 12, further comprising the step 

of: 

providing notification as to the validity of said caller 
routine return address, 

14. The method of claim 12, further comprising the step 

of: 

performing user predetermined acts associated with said 
validity of caller routine address. 

15. The method of claim 12, wherein the step of receiving 
caller routine return address from said process memory 
device further comprises the step of: 
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determining said caller routine calling address by deter- 
mining the address preceding said caller routine of: 
address. 

16. In a computer system running an operating system 
platform, said operating system including a kernel space and 5 
a process space, said process space including a user appli- 
cation running in process space, said user application opera- 
tive to system and function calls, said system or function call 
intercepted, a method of secure function execution, said 
method comprising the steps of: 10 
receiving caller routine return address from said process 

memory device; and 
determining whether caller routine address is valid by 
comparing said caller address routine with associated 
process stack address area. 



17. The method of claim 16, further comprising the step 



providing notification as to the validity of said caller 
routine return address. 

18. The method of claim 16, further comprising the step 

of: 

performing user predetermined acts associated with said 
validity of caller routine address. 

19. The method of claim 16, wherein the step of receiving 
caller routine return address from said process memory 
device further comprises the step of: 

determining said caller routine calling address by deter- 
mining the address preceding said caller routine 
address. 
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